CHAPTER 2	ISIS

I.	INTRODUCTION


	Scene analysis programs have been awkward to develop
using conventional programming systems because of the
difficulty of formulating pictorial descriptions in symbolic
terms.  Picture processing techniques are inherently ad hoc
and must be deduced empirically for each application.  We
have constructed an interactive system specifically
designed for expressing and experimenting with perceptual
strategies.  The system allows an experimenter to describe
basic perceptual concepts to a computer in terms of
pictorial examples.  The examples are designated graphically
by encircling areas of a displayed scene with a cursor.  A
concept is represented internally by values of primitive
feature-extraction operators that distinguish it from
examples of previously defined concepts.  Concepts so
defined constitute a common vocabulary, shared by man and
machine, that can be used symbolically in describing objects
and specifying scene analysis procedures.

	The system has been used to formulate interactively
descriptions that distinguish objects in indoor room scenes
and programs that locate these objects in images.

II.	 DESCRIPTION OF ISIS

	ISIS (Interactive Scene Interpretation System) is an
integrated set of INTERLISP functions facilitating the
development and testing of pictorial representations.  The
system consists primarily of an image file, a library of
primitive feature extraction functions, a means for
applying selected primitive operators to graphically
designated areas of a scene [a symbolic data structure "for
accumulating concept definitions], and an iconic data
structure" for retaining pictorial examples of these
concepts.  

	The symbolic data structure is a semantic net whose
nodes represent prototype patterns for objects, attributes,
and relations.  Prototypes are defined symbolically in terms
of other nodes or iconically by reference to designated
regions of a digitized image.  The system can obtain values
for unspecified properties of a concept by applying
operators to its examples.  Figure 1(a-c) illustrates the
semantic net.  In Figure 1(c), the object TABLETOP and the
attribute BUFF are defined iconically, while the object
FLOOR and the attribute HORIZONTAL are defined symbolically
in terms of other nodes.  FLOOR is distinguished from
TABLETOP by height.

	Iconic regions are represented as an explicit list
of picture samples and/or as a list of vertices describing a
closed polygonal boundary (see Appendix ---).  Efficient routines 
exist for determining whether a given picture element is contained
within a bounded region, for obtaining a set of samples over
a bounded region, and for fitting a boundary around a set of
samples.

	The primitive functions and symbolic data structure
reside within the interactive environment of TENEX INTERLISP
on a PDP-10.  The raw data arrays, and
support routines for graphics, file handling, coordinate
transformations, etc., reside in a FORTRAN environment
accessible as a lower fork through the TENEX operating
system.

	Images are stored in 120 X 120 samples arrays, each
sample characterized by TV brightness through red, green,
blue, and neutral filters and by a range measurement.
(These range measurements are the simulated output of a
time-of-flight range finder, currently under development.) A
high resolution vector display and an MOS-refreshed color
video monitor with a low resolution vector overlay serve as
graphic output devices.  Both displays allow users to select
points on the screen using a cursor.

III.	USE OF ISIS

	To describe concepts in terms of the computer's
primitive functions, a human must know (or be able to
determine easily) what those primitive functions can
distinguish.  Our system provides the trainer with tools
with which to choose primitive functions empirically.  The
system first displays a digitized scene as a color image or
as a threshold gradient.  This display qualitatively conveys
which color boundaries are easily discriminated by the
computer.  Users can use a cursor to draw regions on the displayed 
image and obtain from the system either average or extreme
numerical values for local operators, such as height, hue,
saturation, and surface normal.  Thus, by applying operators
to examples and counter-examples of pictorial concepts in
the image, the trainer can discover directly which operators
provide suffi-cient discrimination.  Users can try out a
proposed description by requesting the system to illuminate
all parts of the displayed scene that correspond to the
description.

********************************************************

EXAMPLE

********************************************************


IV.	DESCRIPTION OF ATTRIBUTES

	Using ISIS, the trainer can define common surface
attributes, such as the color BUFF and the orientation
HORIZONTAL, with which to describe surfaces symbolically.
These attribute labels can usually be defined by a
characteristic range of values for a single primitive
operator.  The range values can be determined empirically by
applying the operator to pictorial examples of the
attribute.  HORIZONTAL, for instance, might be defined as
any surface whose normal is within 5 degrees of the Z axis
based on values of the ISIS function ORIENT applied to image
samples on the FLOOR and TABLETOP.

	The adequacy of a proposed description can be tested
by requesting the system to intensify points in the
displayed image that satisfy a proposed description. This
test is accomplished by using the description to filter
random image samples. In Figure 3, a set of random image
samples is shown superimposed on a gradient display of the
office scene.  In Figure 4, only points satisfying the
suggested definition of HORIZONTAL are retained.  Figure 4
validates the definition of HORIZONTAL, since all
intensified points are on surfaces -- such as the FLOOR,
TABLETOP, and CHAIRSEAT -- normally thought of as HORIZONTAL.

V.	DESCRIPTION OF OBJECTS

	Defined attributes can be used symbolically to
describe objects.  For example, TABLETOP might now be
defined to the system as a HORIZONTAL surface distinguished
by height.  The appropriate height range was determined by
applying the ISIS function HEIGHT to a few manually
designated TABLETOP sample.  Figure 5 confirms that height
and orientation adequately distinguish TABLETOP in this
particular scene.

	It is not always possible to specify a simple
predicate that will select all image samples belonging to a
desired object and to no others.  For such cases, the
repertoire of ISIS primitives described in this summary can
be augmented with special purpose feature extraction
routines.  Procedures which validate edges, grow regions and
bound rectangular surfaces have been developed by several
users and are now generally available on ISIS.  

VI.	EXTENSIONS of INTERACTIVE SCENE ANALYSIS

	Ideally, one would like to program a computer to
find an unfamiliar object as one would instruct a person, by
providing a crude description of the desired object. A tree,
for example, might be described as a "green, leafy region
above a tall, brown, vertical, bark-textured region." The
computer could demonstrate comprehension by outlining
instances of the described object in a displayed image.  The
programmer could then refine the description empirically to
correct errors in the computer's interpretation.

	Communication on this level with a machine is
hampered because the machine does not share an
"understanding" with the human of basic descriptive concepts
like "green," "leafy," "above," "vertical," and
"bark-textured." Such concepts are often difficult to
express in natural language, and even more so in
conventional programming languages.  Our aim has therefore
been to create an interactive system that allows an
experimenter to describe basic perceptual concepts to the
computer using pictorial examples.  The examples are
designated graphically by encircling portions of a displayed
scene with a cursor.  Given a pictorial example, a
representation can be empirically constructed by trying
pictorial operators in order of increasing complexity until
the example is sufficiently distinguished from previously
determined representations. Pictorially defined concepts
constitute a shared vocabulary that can be used to describe
objects or to define more complex concepts.  This paradigm
is intended to elevate the user above the details of
hand-coded algorithms, allowing him instead to concentrate
on the construction of descriptions and strategies.